• Add CPU hog scenario functional tests with ephemeral namespace isolation
• Implement test cases for success lifecycle, node selector, duration validation
• Add failure scenarios for invalid node selector and malformed YAML
• Create base scenario configuration for CPU hog stress testing

CPU hog scenario functional test suite implementation

• Implement four functional test cases for CPU hog scenario execution
• Test pod lifecycle: creation during run and cleanup after completion
• Validate node selector targeting and duration constraints
• Verify graceful failure handling for invalid configurations
• Add helper functions to poll and retrieve hog pods from namespace

CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py

Add cpu_hog pytest marker configuration

• Add cpu_hog pytest marker for CPU hog scenario test identification
• Enable selective test execution and filtering by scenario type

CI/tests_v2/pytest.ini

Base CPU hog scenario configuration template

• Define base CPU hog scenario configuration with default parameters
• Set duration, worker count, CPU load percentage, and image reference
• Configure namespace and node selector targeting for test execution
• Specify hog type as CPU with all CPU methods and single node target

CI/tests_v2/scenarios/cpu_hog/scenario_base.yaml

Description

test_cpu_hog_success_and_lifecycle calls proc.communicate(timeout=90) without handling subprocess
timeout, so a slow/hung kraken run will raise and leave the background process running (with
stdout/stderr pipes) and can hang the overall test session.

Code

CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py[R56-64]

+        proc = self.run_kraken_background(config_path)
+        try:
+            pods = _wait_for_hog_pod(
+                self.k8s_core, ns, self.HOG_POD_PREFIX, timeout=POLICY_WAIT_TIMEOUT
+            )
+            assert len(pods) >= 1, f"Expected at least one hog pod in namespace={ns}"
+        finally:
+            # duration=10 + pod wait (30s) + cleanup; allow 90s for Krkn to exit.
+            stdout, stderr = proc.communicate(timeout=90)

Evidence

The test starts kraken using a Popen with stdout/stderr pipes and then unconditionally calls
communicate(timeout=90) in a finally block without any termination/kill fallback, so a
TimeoutExpired will abort cleanup and can leave the subprocess alive. The suite already defines
overridable timeout constants (including KRAKEN_PROC_WAIT_TIMEOUT), but this test hard-codes 90
seconds, making behavior inconsistent when env overrides are used.

CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py[45-65]
CI/tests_v2/lib/kraken.py[44-58]
CI/tests_v2/lib/base.py[38-46]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`test_cpu_hog_success_and_lifecycle` starts Kraken in the background and calls `proc.communicate(timeout=90)` without handling `subprocess.TimeoutExpired`. If Kraken hangs or runs longer than expected, the test errors out and can leave the subprocess running (with stdout/stderr pipes), which may stall the test session and leak cluster resources.

### Issue Context
- `run_kraken_background` uses `stdout=PIPE` and `stderr=PIPE`.
- The test suite already defines configurable timeout constants (`KRAKEN_PROC_WAIT_TIMEOUT`, `TIMEOUT_BUDGET`, etc.) via env vars.

### Fix Focus Areas
- CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py[56-69]
- CI/tests_v2/lib/base.py[38-46]

### Suggested implementation notes
- Wrap `proc.communicate(...)` in `try/except subprocess.TimeoutExpired`.
- On timeout: `proc.terminate()` then `proc.kill()` if still running; drain output; then fail with a clear message.
- Replace the hard-coded `90` with `KRAKEN_PROC_WAIT_TIMEOUT` or a computed timeout derived from `scenario['duration']` plus a buffer.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

If no hog pod appears and _wait_for_hog_pod raises TimeoutError, the test never reaches the code
that wraps stdout/stderr and calls assert_kraken_success, so the kraken output captured in the
finally block is not persisted or shown, making CI failures hard to debug.

Code

CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py[R56-70]

+        proc = self.run_kraken_background(config_path)
+        try:
+            pods = _wait_for_hog_pod(
+                self.k8s_core, ns, self.HOG_POD_PREFIX, timeout=POLICY_WAIT_TIMEOUT
+            )
+            assert len(pods) >= 1, f"Expected at least one hog pod in namespace={ns}"
+        finally:
+            # duration=10 + pod wait (30s) + cleanup; allow 90s for Krkn to exit.
+            stdout, stderr = proc.communicate(timeout=90)
+        result = SimpleNamespace(
+            returncode=proc.returncode,
+            stdout=stdout or "",
+            stderr=stderr or "",
+        )
+        assert_kraken_success(result, context=f"namespace={ns}", tmp_path=self.tmp_path)

Evidence

The test waits for hog pods inside a try and then builds the result object and calls
assert_kraken_success only after the try/finally. Any exception from _wait_for_hog_pod exits the
test before the result/assertion path, and unlike assert_kraken_success (which writes logs to
tmp_path on failure), there is no log persistence in this exception path.

CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py[56-70]
CI/tests_v2/lib/utils.py[166-188]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
When `_wait_for_hog_pod(...)` times out, the test raises `TimeoutError` before it creates the `result` object and calls `assert_kraken_success`. Although `stdout/stderr` are obtained in the `finally` block, they are not persisted or surfaced on this exception path.

### Issue Context
`assert_kraken_success`/`assert_kraken_failure` already have a convention of writing `kraken_stdout.log` / `kraken_stderr.log` to `tmp_path`, but this path bypasses those helpers.

### Fix Focus Areas
- CI/tests_v2/scenarios/cpu_hog/test_cpu_hog.py[56-75]
- CI/tests_v2/lib/utils.py[166-188]

### Suggested implementation notes
- Capture exceptions from the pod-wait block (e.g., `except Exception as e: exc = e`) and in `finally` write `stdout/stderr` to `tmp_path` before re-raising/failing.
- Alternatively, convert the timeout into an `AssertionError` that includes the last N lines of kraken stdout/stderr and points to the log files.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools